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ABSTRACT 

When art and technology meet, a huge information flow has to be 
managed. The LISTEN project conducted by the Fraunhofer Institut in St. Augustin 
(Germany) augments every day environments with audio information. In order to 
distribute and administer this information in an efficient way, the Institute decided 
to employ an information brokering tool for the management of information items. The 
generation of user profiles and the personalized presentation of information are 
possible by this means. This paper depicts an approach of transferring this 
information brokering experience to Web museum applications. It shows how the LISTEN 
domain model can’ easily be extended by an overlay model and adapted to a Web museum 
environment. (Contains 12 references.) (Author) 
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Abstract 



When art and technology meet, a huge information flow has to be 
managed. In the LISTEN project conducted by the Fraunhofer Institut in 
St. Augustin, we augment every day environments with audio information. 

In order to distribute and administer this information in an efficient way, we 
decided to employ an information brokering tool for the management of 
information items. Furthermore, the generation of user profiles and the 
personalized presentation of information are possible by this means. This 
contribution depicts an approach of transferring our information brokering 
experience to Web museum applications. We show how the LISTEN 
domain model can easily be extended by an overlay model and adapted to / 
a Web museum environment. 
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Introduction 

A variety of guidance systems or information systems have been developed in 
the last few years for the support of a museum visit or for preparation for such a 
visit. In most cases the authenticity and the possibility of contextualizing the 
information presentation to the current position or situation of a user were seen 
as central issues. Furthermore, classical museum audio guides were much more 
acceptable to museum visitors because of the easy handling and the quality of 
sound presentation. For the flexible use of information about art objects and 
museum exhibitions, we propose a centralized model around a domain ontology 
that is described in a software tool for information brokering. We think that the 
proposed model is a very flexible way to reuse existing information and support 
curators and exhibition experts to in the design of a variety of personalized 
museum experiences, ranging from an exhibition Web site to a interactive audio 
experience in the museum space to a personalized CD-ROM production for 
taking home. 

Context as a mean for adaptation of information selection and presentation has 
been described in a variety of ways and approaches (Gross & Specht, 2001; 

Shilit, Adams & Want, 1994; Dey & Abowd, 1999). Nevertheless the underlying 
problem of identifying similarities and differences between various constellations 
of context parameters has not been discussed intensively in the literature. From 
our point of view identifying important context parameters to describe user 
interaction is an essential issue when designing context aware information 
services. 

The identification and the description of the context parameters is a non-trivial 
task. A central issue is to find a solution for structuring an information domain 
appropriate not only from the point of view of an information engineer but also 
from the point of view of a user. Especially for designing personalized information 
services the structure and the intuitiveness of the information structuring is 
essential for the successful application of user modeling and personalization 
methods. 

BEST COPY AVAN ARLF 

->2 



file://E:\mw2003\papers\zimmermann\zimmermann.html 



5/27/2003 






The LISTEN project conducted by the Fraunhofer Institut in St. Augustin deals 
with the audio augmentation of real and virtual environments. A first LISTEN 
prototype has been successfully installed at the Kunstmuseum in Bonn (Unnltzer, 
2001). Visitors walk through the museum and experience personalized audio 
information about paintings by August Macke through wireless headphones. In 
such a scenario, the distribution and authoring of this valuable information on 
exhibits is a non-trivial task. 

Audio-Augmenting Reai and Virtuai Environments 

Combining high-definition spatial audio rendering technology with advanced user 
modeling methods creates audio-augmented environments. Visitors are 
immersed in a dynamic virtual auditory scene that consistently augments the real 
space they are exploring. The physical environment is augmented through a 
dynamic soundscape, which users experience over motion-tracked wireless 
headphones for 3D spatial reproduction of the virtual auditory scene. 

A sophisticated auditory rendering process takes into account the current position 
and orientation of the visitor's head in order to seamlessly integrate the virtual 
scene with the real one. Speech, music and sound effects are dynamically 
arranged to form an individualized and situated soundscape offering exhibit- 
related information as well as creating context-specific atmospheres. Next to the 
automatic adaptation of sound scene rendering to the position and orientation of 
the user's head, the audio stream is controlled in two ways: events (mediated 
interaction) that are used to start and stop the playback of information items in 
form of audio recordings; and continuous control (immediate interaction) 
changing parameters in the audio-generation of the presentation (e.g. a sound 
that gets continuously louder as you approach a certain position within the 
space). 

The dynamic composition of the soundscape is personalized through each 
visitor's spatial behavior, the history of the visit, and interests or preferences 
either expressed explicitly by the visitor or inferred from the visitor's behavior 
(Eckel, 2001). To present individualized media and create augmented 
environments, such systems have several models in common (Gossmann & 
Specht, 2001 ) that are described in more detail in this section: the World model, 
the augmentation layer, the domain model and the user model. 

The World Model (Space Model , Location Model) describes the physical 
environment the user moves through while interacting with the system. In the 
LISTEN environment, the space model contains the geometric information of the 
exhibition space and its objects. The LISTEN world model is a detailed VR-based 
geometric model. It is created for the AVANGO application (Tramberend, 1999) 
and is described as a geometric scene graph. Therefore, a LISTEN environment 
can be tested and prototyped in a CAVE system (Eckel, 2001), or be explored in 
real space with virtual audio content displayed through a wireless motion-tracked 
headphone. 

The Augmentation Layer on top of the World Model defines areas (Zones, 
Segments, Triggers) within the world model that contain active information or 
sound objects the users of the system interact with. The augmentation layer 
filters the position and motion of the user by dividing the dimensions the user 
moves through (location and orientation) into meaningful constraints and deriving 
continuous parameters from them. By defining zones and segments, the visitor's 
focus obtains a valuable meaning. 

The Domain Model holds information about sound objects and other hypermedia 
objects connected to the physical space via the augmentation layer by using 
meta data. The domain model builds up a virtual acoustic space in which the 
location of virtual sound sources and spaces are defined. Stopping in front of an 
exhibit generates aural information about the art piece. Moving the head and 
body activates a further audio source, where music deepens the visitor's 
impressions, or the voice of a commentator talks about the artist or describes the 
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period the painting originates from. 



The User Model contains knowledge and profile information about the system’s 
users. While the user moves in physical space, events are sent to the user 
model, and by these events the model is refined. The user model stores 
knowledge about the visitor - like preferences, interests in arts, and the history of 
the visit. In combination with the visitor’s spatial position (delivered by the 
tracking system), the content of the user model strongly influences the 
presentation of information according to the current visitor's context. 

Information Brokering 

In this real world application of LISTEN, we decided to use an information 
brokering tool for authoring valuable information on exhibits and distributing 
requested information among the visitors in an efficient way. Information 
brokering is a value adding process of mediation between information demands 
and information offers. The added value emerges from the understanding of the 
domain complexity and the definition of a useful vocabulary. Additional structures 
and interpretation rules (implicit and explicit) in information exchange simplify the 
processing and comprehension of exchanged information. Figure 1 illustrates the 
interrelation between data, information, and knowledge. 




Three roles participate in the information brokering process: the provider offers 
information, the consumer demands information, and the broker mediates 
between the other two. The quality of the information brokering process depends 
to a great extent on the knowledge available to the broker. Knowledge about 
available sources, the domain, consumers and other brokers is needed. Source 
knowledge is created in the domain representation or maintenance processes 
and describes the quality of sources and how they can be accessed. Domain 
Knowledge is about the contents of the brokering domain and should reflect the 
provider’s understanding of the domain as well as the consumers’ perception of 
the domain. Consumer Knowledge is created in the consumer-oriented process 
and describes the consumer and his concrete information need. Consumer 
knowledge has to map onto domain knowledge to fulfill the information need. To 
ensure an optimal service to consumers, they should be served by the best 
broker according to their information need. This assignment task depends on the 
availability of Expert Knowledge about different brokers. 

In the LISTEN project, the brokering tool can be understood as a server 
mediating information items between the exhibition's curator and the LISTEN 
application. In the role of the information provider, the curator defines the domain 
model and authors the information items, as illustrated in the left column of Figure 
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2. From the point of view of the information broker, the information is consumed 
by the LISTEN application, although it finally passes the information to the visitor 
as the real consumer. In the role of the information consumer (Figure 2, right 
column), the LISTEN application defines an interest model regarding to the 
visitor’s location, preferences, interests, and history of already visited objects. 

To mediate between the provider and the consumer, a brokerage tool acts as a 
server to deliver filtered information on demand. Such a brokerage development 
and management environment (the Broker's Lounge ; Jarke, Klemke & Nick, 

2001) has been developed at the Fraunhofer Institut for Applied Information 
Technologies (FIT) in St. Augustin. With the aid of this application, a large variety 
of scenarios within the general framework of Figure 1 and Figure 2 can be quickly 
developed, and efficiently and flexibly executed. 




Figure 2 

Information brokering processes make use of and create a number of information 
items that describe single units of information (cf. Klemke & Koenemann, 1999). 
As Klemke (2002) has defined, each information item is an instantiation of a 
concept which describes the structure of the brokered items. In order to organize 
information items, categories describe fundamental principles or ideas. In many 
cases, these categories define hierarchical trees. In Figure 1 conceptualization 
and categorization is illustrated as a procedure for the concretion of raw data. 

The next section provides an outline of definition of the LISTEN project’s 
information items and illustrates a hierarchical tree to categorize them. 

Modeling Information Items 

Bringing arts and technology together means the administration and distribution 
of a huge amount of information. In this paper we propose the use of a central 
management component for all information items. We present an approach of 
modeling information items in a museums environment by means of domain 
ontology in an information brokering tool. In the case of the LISTEN project, the 
set of information items is composed of sound items. We describe the application 
of the ontology methodology (concepts and categories) to generate a meaningful 
domain model and depict the association of information items to exhibition 
objects. 

In recent projects developing guiding systems and electronic art guides, the 
structuring and internal representation of information appeared to be a central 
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issue for delivering the right information at the right time to a specific user. Even 
more, for the personalization of presentations and the contextualized delivery of 
information pieces, the underlying structure of the represented information 
needed to be a core part of the work. Nevertheless there was a trade off between 
the efforts of authoring information items into a highly enriched information 
representation and the daily work of curators and information providers for 
museum environments. 

To find the right balance between user efforts for authoring and creating museum 
information and meta data in previous projects, we often used an object-oriented 
approach, where the main entities were the art objects as such. This led us to 
typical presentation and structuring of information as found in art databases and 
museum information systems. 
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Figure 3 

By detailed user needs analysis of museum visitors and by the analysis of human 
guides, we realized that the event character of museum visits in most cases is 
much more important than plain information about the art object as such. The 
presentation of data about artworks is just one style of presentation that can be 
appropriate but in practice is rarely used by human guides. Furthermore, through 
our work with museum curators and artists in the LISTEN workshops we elicited 
different strategies and methodologies to structure information items and to 
combine those information items in a highly flexible way for different users. 

The main entities for generating presentations are therefore not the artworks as 
such, but the parts or chunks of a presentation that can come from a diversity of 
sources and use a variety of stylistic means to make the museum visit an 
interactive experience. Therefore, for creating interactive audio augmented 
LISTEN spaces, we decided to choose the sound item as the main entity. Sound 
items can be classified in a category system with several dimensions describing 
the sound items technically and stylistically. An overview of the dimensions can 
be seen in Figure 3. This figure illustrates the domain model inside the graphical 
interface of the information brokering tool, organized as a tree, and a list of 
information items of the LISTEN project. 

The single sound items as such are independent episodes or chunks of 
presentation that can be combined flexibly because they contribute to a variety of 
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art objects. The expressive power of such a structure can be easily explained 
with an example: 

In an introduction to several artworks of August Macke, the different artworks are 
connected by personal episodes, events and experiences of August Macke, by 
reactions and interaction with his social environment (personal letters, media, 
press articles), or by the “Zeitgeist" and other factors of the environment of 
August Macke at that time. In most cases human guides extend the visual 
perception of such artworks with these "stories or impressions” of that time period 
to immerse the user in an authentic experience of the works and the life of the 
artist. By the structured description (meta tagging) of a huge collection of such 
“stories or impressions," the LISTEN system allows a highly flexible way of 
"immersing the user" into the August Macke experience. 

Therefore we designed the domain ontology (meta data) for the Macke-Exhibition 
to allow for the description of small information items on a variety of dimensions, 
allowing for the connection and the individualized sequencing and presentation of 
these items. Beside the simple classification of the information items in a tree 
structure, the information brokering tool allows for classification of sound items 
into multiple categories. 

The domain ontology contains: 

• technical descriptions of the sound items; such as length of item, type 
(music, speech, sound effects) 

• classification concerning the relation to the physical space objects (art 
objects to which an item contributes, physical area zones or focuses to 
which they are connected) 

• classification about phases of work, image genre, or art technical aspects 

• classification about the preferred target group, like the stereotypical 
listeners for such a sound item, or the emotional impacts or dramaturgy. 

Especially speech sound items could be further classified into subcategories like 
Citation, Collage, Diary, Letter, Newspaper and others to describe their style of 
presentation. As described in the example of the Macke-Exhibition, the 
multidimensional classification of the sound items allows for a variety of 
sequences and presentation styles, even combining sound items on several 
channels; i.e. typically music, effects, and speech. An enormous advantage of the 
description in such away is that the curator of an interactive experience is not 
forced to design a complete sequence of information presentation but can 
combine resources in a collage style or define sequencing rules on the level of 
possible connection categories and not for single sound items. 

In the LISTEN application, the visitor's position and orientation are observed by a 
tracking system and interpreted within the context of a virtual environment, which 
connects the real world objects with virtual objects. Since the museum's visitors 
are mobile in physical space, their spatial positions are translated into virtual 
positions in the electronic space relative to virtual objects. The virtual 
environment enables the definition of virtual sound sources and the segmentation 
of the physical space into virtual zones. Invisible to the user there exists a 
specific category in the broker’s domain model for each exhibition object. 
Depending on the visitor’s location, preferences and so forth, combinations of 
such categories are dynamically selected in order to pre-filter the information 
items. The next section illustrates how this solution can be adapted to an 
electronic museum. 

Adaptation to a Web Museum 

In order to transfer our information brokering solution to the domain of an 
electronic museum, we present a modus operandi for mapping the physical 
space into an electronic space. For this reason we follow an overlay model 
approach for context-aware information systems as introduced by Gross & 

Specht (2001). This methodology enables us to apply and reuse the algorithm for 
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information filtering described in the previous section within another domain. 

Thus, only three concepts of the LISTEN application must be adapted or 
extended: tracking, modeling of the information items, and domain modeling. 

As described in the last section, the domain model connects the objects to 
categories. If the user of the LISTEN system enters the zone connected with a 
certain domain object, the LISTEN application selects the category associated 
with this object, in order to filter information items according to the users 
preferences and position. In information brokering terminology, the application 
builds an information model and uses the broker as triggered by events. In an 
electronic museum application, the tracking system would deliver the current URI 
as position and the selected objects (e.g. text links, images, or clicks on maps) as 
a kind of the visitor’s movement. Here, an event is fired when the user visits any 
Web source that is connected with a category of the broker's domain model. 



The information architecture of the LISTEN system is limited to audio 
presentations. The information broker falls back on a set of audio pieces (e.g. 
spoken text, music, effects) and returns the best matching item according to the 
inquired interest model of the user. In the domain of an electronic museum, the 
curator may probably extend this set of information items by any kind of 
multimedia presentations, text fragments or links to Web pages. Therefore, the 
broker's domain model must be extended also to adequately reflect the changed 
information architecture. New categories have to be added and the meta data 
description has to be augmented, easily accomplished with the assistance of the 
information brokering tool. As a result, the system provides additional facility for 
specification of preferences to the users (e.g. users can turn off streamed video- 
presentations), and thereby the user model is extended at the same time. 

In contrast to the fixed sound installation in the real world LISTEN environment, 
an electronic museum has to take into account the abilities and features of the 
visitor's particular environment; i.e. the Web browser. An initially executed piece 
of software can ask for the user’s specific settings and automatically preset the 
information brokering tool. Based on known environmental properties, the 
information items are filtered in advance and presented in an appropriate format. 
The ontology model of the information brokering tool has to be adapted in a 
suitable way. 

The mentioned extensions of the information brokering tool do not affect its 
capability of filtering information items that fit best the user's context. The overlay 
model enables the retention of the presented concepts no matter whether the 
user moves in physical or in electronic space. The next section shows how the 
presentation of information and exhibits in an electronic museum can further be 
personalized and adapted to the visitor's needs and behavior. By means of a 
user model, additional filters can be triggered or refined. 

Personalization of a Web Museum 

A Web museum environment offers the ability to adapt the order of the presented 
exhibits and the attached information to the visitor's profile and position within the 
exhibition. In order to provide a personalized adaptation of the environment 
according to the visitor's context (i.e. interests, preferences and “motion”), the 
system has to build up and maintain a user model. 

By using sensor data, the user is tracked in the information space in the same 
way a location tracking system tracks users in physical space. This allows for a 
variety of new applications and is also an important data resource in user 
modeling and adaptive systems. Basically speaking, the growing number of data 
resources about a user allow more valid inferences and much more 
contextualized interactions between users and adaptive systems. 

The enrichment of information items with significant meta information enables the 
personalization and customization of information offers. By requesting user 
preferences .different user profiles can be built up to facilitate information filtering 
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according to the user’s needs. Besides information presentation, the system can 
provide recommendations to the visitor regarding context. These recommended 
exhibition objects may attract the visitor’s attention by emitting an attractor cue 
(so-called “prompting” of the user). On the basis of meaningful user profiles, 
several adaptive strategies can be applied to guide the visitor to the Web 
museum on a specific (strictly predefined or context-aware adapted) tour. 

In our approach, the personalization process is divided into four steps: 
information collection, modeling, controlling and rendering. Each step fulfils a 
certain role within the user modeling process. The next subsections describe 
these modules in more detail. 

Information Collection 

A network of sensors is placed in the environment and connected to variable 
parameters of the domain. These sensors are used for recognizing changes 
within the environment, and especially for the perception of the user’s interaction 
with this environment. An observation module receives all incoming events sent 
by the Web-Server. These event descriptions are pushed into a database. Thus, 
an event history for every visitor is saved, and an implicit user profile is recorded. 

Modeling 

By the means of statistical models, the implicit user profile already allows the 
deduction of valuable information that can be used for standard adaptation 
activity (e.g. the more time the visitor spends with the art exhibit, the more s/he 
likes it). This deducted information builds a behavior model of the visitor. The 
behavior model can be treated like explicit representation of implicit user 
feedback and may be consulted to draw an assumption about the user’s interest 
in a specific object. It is planned to gain more significant information relating to 
the behavior of the user by implementing different machine learning and data 
mining algorithms to extract semantically enriched information. 

In our user modeling approach for a museum environment, we chose to employ 
our above mentioned adapted information-brokering tool for modeling user 
preferences. Parts of the domain model of the museum environment are mapped 
to an ontology model (as shown in Figure 3). The users specify their preferences 
by simply selecting topics they are interested in from the displayed categories. 
From these requests different visitor profiles can be built up and stored within the 
tool. Based on these user selections and on significant meta data describing 
information items, the personalization process performs a pre-selection and pre- 
filtering of information offers and customizes according to the users’ needs (cf. 
Pazzani & Billsus, 1997). 

The utilization of stereotypes is common in adaptive systems. With the aid of 
stereotypes, the personalization engine defines the observation type of a visitor, 
and thus, the system is able to adapt the scenery accordingly. Every stereotype 
causes a different presentation and different strategies (cf. next subsection). 
These stereotypes are also part of the ontology model of the mentioned 
information-brokering tool. At the moment, the visitor’s classification into one 
stereotype is done manually by that visitor and cannot be changed automatically 
during runtime. Additionally, the personalization engine is not able to perform an 
automated clustering of user profiles and derive new stereotypes from this 
process. 

Controlling 

Meaningful user profiles accurately document visitor activity within the museum 
and can be exploited to adapt the environment in order to support the visitor, to 
provide personalized information or to invest the exhibition with an even more 
artistic touch. Therefore, a controlling component is necessary to decide what 
consequences must occur if certain conditions in the visitor’s environment and in 
the individual user model configuration appear together. Based on these 
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information sources ,the control layer assembles a sequence of commands in 
order to adjust certain variable properties of the environment. Thus, different 
sequences of commands lead to different kinds of information presentation. This 
adjustment of environmental parameters is used for realizing domain 
independent adaptive or strategic methods, and domain dependant expressive 
methods. Examples for domain independent adaptation methods on a strategic 
level are adaptive prompting and adaptive annotation for objects. Expressive 
methods that seem to be appropriate for the personalization of Web museums 
are the following: 

/ 

Adaptation of the presentation 

The basis for every kind of adaptation is the presentation of the exhibits with 
some associated information. Besides the decision about which item is to be 
shown, the ways presentation can be modified are manifold, and with 
combinations of these possibilities, a wide range of adaptability is already 
accomplished. 

Adaptation to social context 

If visitors are spatially and temporally similar (e.g. two visitors looking at the same 
exhibit), they may obtain similar information or may be brought together for a 
chat. Through building such clusters of people, for example, a subsequent 
discussion about seen objects is possible (cf. Zimmermann, Lorenz & Specht, 
2002). 

Adaptation to the level of “immersiveness” 

Within a museum environment, interest in objects may be expressed by the time 
a visitor’s focus lingers on these objects. The level of interest corresponds to the 
complexity, the amount, and the style of already received information about one 
object and is transferred to succeeding objects. If one of these objects complies 
with the visitor’s interests, the presentation style directly steps into the right level 
of interest, and information items that are classified at the adequate information 
depth and style are displayed. 

Adaptation to movement and perception styles 

Several kinds of common behavior can be identified with people “moving” through 
the environment (e.g. clockwise in real museums). Attractor cues (e.g. sounds, 
blinking or marked links) emitted from different sources are used to draw the 
user’s attention to certain objects in the environment. Thus, entire tours through 
the Web museum can be recommended. The selection and dynamic adaptation 
of tour recommendations can be adjusted to the visitor's stereotypical type of 
movement and preferred perception style. 

Rendering 

Rendering means handling the connection back to the domain. This engine 
translates the assembled sequence of domain-independent commands into 
domain-dependent commands. The implemented domain-dependent methods 
directly change variable parameters of the domain (i.e. content of a HTML page) 
according to the user's behavior. Thus, the decisions taken by the controlling 
component are to be mapped to real world actions. 

Conclusion 

Information brokers mediate between information demands and information 
offers. The information items to be brokered in the LISTEN system are sound 
pieces the users experience during their movements through everyday 
environments. The use of an information brokering tool facilitates the 
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management and distribution of a huge amount of information within this domain. 
Through mapping domain properties to an ontology model, we benefit from a 
better understanding of the domain’s complexity. In addition, the ontology model, 
in combination with the enrichment of information items with meta data 
descriptions, enables a personalized presentation of audio information. 

In this contribution we presented a concept for reusing our experience, gained 
during the application of the information brokering tool within audio-augmented 
environments, in the context of a Web museum. We followed an overlay model 
approach that builds on and extends the LISTEN methodology in three ways: 
tracking, information item modeling ,and domain modeling. The information 
brokering tool supports the adaptation of these three concepts, so that an 
electronic museum may take advantage of the benefits. 
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